Change %28Detection%29 You Can Believe in Finding Distributional Shifts in Data Streams.

Database Exploration Using Database Dynamics.

Information quality for network monitoring and analysis.

Database Exploration Using Database Dynamics.

Mining Network Logs Information Quality Challenges.

Stewardship and Staging of Information for Enterprise Mining.

Exploratory Data Mining and Data Cleaning

Data quality through knowledge engineering.

Data Quality and Data Cleaning An Overview.

Database Exploration and Bellman.
Large industrial-scale databases tend to be poorly structured, dirty, and very confusing. There are many reasons for this disorder, not the least of which is that the application domains themselves are poorly structured, dirty and confusing. As data analysts, we are often called upon to mine, clean, or otherwise analyze these databases. In this article, we describe the types of problems we have encountered, tools and techniques we have developed to address these problems, and directions for future work.
Mining database structure%3B or%2C how to build a data quality browser.
ABSTRACT Data mining research typically assumes that the data to be analyzed has been identified, gathered, cleaned, and processed into a convenient form. While data mining tools greatly enhance the ability of the analyst to make datadriven discoveries, most of the time spent in performing an analysis is spent in data identification, gathering, cleaning and processing the data. Similarly, schema mapping tools have been developed to help automate the task of using legacy or federated data sources for a new purpose, but assume that the structure of the data sources is well understood. However the data sets to be federated may come from dozens of databases containing thousands of tables and tens of thousands of fields, with little reliable documentation about primary keys or foreign keys. We are developing a system, Bellman, which performs data mining on the structure of the database. In this paper, we present techniques for quickly identifying which fields have similar values, identifying join paths, estimating join directions and sizes, and identifying structures in the database. The results of the database structure mining allow the analyst to make sense of the database content. This information can be used to e.g., prepare data for data mining, find foreign key joins for schema mapping, or identify steps to be taken to prevent the database from collapsing under the weight of its complexity. 1.
A Data Quality Browser.
ABSTRACT Data mining research typically assumes that the data to be analyzed has been identified, gathered, cleaned, and processed into a convenient form. While data mining tools greatly enhance the ability of the analyst to make datadriven discoveries, most of the time spent in performing an analysis is spent in data identification, gathering, cleaning and processing the data. Similarly, schema mapping tools have been developed to help automate the task of using legacy or federated data sources for a new purpose, but assume that the structure of the data sources is well understood. However the data sets to be federated may come from dozens of databases containing thousands of tables and tens of thousands of fields, with little reliable documentation about primary keys or foreign keys. We are developing a system, Bellman, which performs data mining on the structure of the database. In this paper, we present techniques for quickly identifying which fields have similar values, identifying join paths, estimating join directions and sizes, and identifying structures in the database. The results of the database structure mining allow the analyst to make sense of the database content. This information can be used to e.g., prepare data for data mining, find foreign key joins for schema mapping, or identify steps to be taken to prevent the database from collapsing under the weight of its complexity. 1.
Data Quality Issues in Service Provisioning %26 Billing.

Hunting Down Glitches in Massive Time Series Data.

Hunting of the Snark Finding Data Glitches using Data Mining Methods.
Data quality is critical to data analysis because bad data can lead to incorrect conclusions. Problems  with data are best detected early, before too much time and effort are spent ingesting and analyzing  it. In this paper, we propose the use of data mining techniques for the automatic detection of data  problems commonly encountered in large multivariate data sets. Data mining methods are ideal for this  purpose, since they are designed for finding abnormal patterns in large volumes of data. We discuss  some important types of data integrity issues. We demonstrate the use of a data mining method, the  DataSphere set comparison technique (from our earlier work [6]) to detect glitches that mimic the error  conditions discussed, using artificial data.  1 Introduction  Corporations base important decisions on results derived from data. Which segments of the customer base are growing? What kinds of products are they buying? What are the correlations between the products bought? Is the inf...
Comparing Massive High%2DDimensional Data Sets.
The comparison of two data sets can reveal a great  deal of information about the time-varying nature of an  observed process. For example, suppose that the points  in a data set represent a customer&#039;s activity by their  location in n-dimensional space. A comparison of the  distribution of points in two such data sets can indicate  how the customer activity has changed between the  observation periods. Other applications include data  integrity checking. An unexpected change in a data set  can indicate a problem in the data collection process.  We propose a fast, inexpensive method for comparing  massive high dimensional data sets that does not make  any distributional assumptions. The method adapts  the power of classical statistics for use on complex,  high dimensional data sets. We generate a map of  the data set (a DataSphere), and compare data sets  by comparing their DataSpheres. The DataSphere can  be generated in two passes over the data set, stored  in a database, and aggregat...
